Ordered index

ABSTRACT

Systems and methods for processing an index are described. A postings list of items containing a particular term are ordered in a desired retrieval order, e.g., most recent first. The ordered items are inserted into an inverted index in the desired retrieval order, resulting in an ordered inverted index from which items may be efficiently retrieved in the desired retrieval order. During retrieval, items may first be retrieved from a live index, and the retrieved items from the live and ordered indexes may be merged. The retrieved items may also be filtered in accordance with the items&#39; file grouping parameters.

BACKGROUND

Modern data processing systems, such as general purpose computersystems, allow the users of such systems to create a variety ofdifferent types of data files. For example, a typical user of a dataprocessing system may create text files with a word processing programsuch as Microsoft Word or may create an image file with an imageprocessing program such as Adobe's PhotoShop. Numerous other types offiles are capable of being created or modified, edited, and otherwiseused by one or more users for a typical data processing system. Thelarge number of the different types of files that can be created ormodified can present a challenge to a typical user who is seeking tofind a particular file which has been created.

Modern data processing systems often include a file management systemwhich allows a user to place files in various directories orsubdirectories (e.g. folders) and allows a user to give the file a name.Further, these file management systems often allow a user to find a fileby searching not only the content of a file, but also by searching forthe file's name, or the date of creation, or the date of modification,or the type of file. An example of such a file management system is theFinder program which operates on Macintosh computers from AppleComputer, Inc. of Cupertino, Calif. Another example of a file managementsystem program is the Windows Explorer program which operates on theWindows operating system from Microsoft Corporation of Redmond, Wash.Both the Finder program and the Windows Explorer program include a findcommand which allows a user to search for files by various criteriaincluding a file name or a date of creation or a date of modification orthe type of file. This search capability searches through informationwhich is the same for each file, regardless of the type of file. Thus,for example, the searchable data for a Microsoft Word file is the sameas the searchable data for an Adobe PhotoShop file, and this datatypically includes the file name, the type of file, the date ofcreation, the date of last modification, the size of the file andcertain other parameters which may be maintained for the file by thefile management system.

Certain presently existing application programs allow a user to maintaindata about a particular file. This data about a particular file may beconsidered metadata because it is data about other data. This metadatafor a particular file may include information about the author of afile, a summary of the document, and various other types of information.Some file management systems, such as the Finder program, allow users tofind a file by searching through the metadata.

In a typical system, the various content, file, and metadata are indexedfor later retrieval using a program such as the Finder program, in whatis commonly referred to as an inverted index. For example, an invertedindex might contain a list of references to documents in which aparticular word appears. Given the large numbers of words and documentsin which the words may appear, an inverted index can be extremely large.The size of an index presents many challenges in processing and storingthe index, such as updating the index or using the index to perform asearch.

SUMMARY OF THE DETAILED DESCRIPTION

Methods and systems for processing an inverted index in a dataprocessing system are described herein.

According to one aspect of the invention, a method for preparing anordered inverted index is described that, among other advantages, allowsfor faster retrieval of items from the index. While scanning items toindex the terms that the items contain, a postings list of itemscontaining a particular term are ordered in a desired retrieval order.The ordered items are inserted from the postings list into an index inthe in the desired retrieval order to create an ordered inverted index.During retrieval, items from the ordered inverted index are returned inthe desired retrieval order more quickly and efficiently than if theywere retrieved from a conventional unordered index.

According to one aspect of the invention, additional efficiency inretrieving items in the desired retrieval order may be realized bygenerating a second inverted index separate from the ordered invertedindex, where the second inverted index is generated from live items,i.e., recently created or updated items that represent current, or“live,” updates to the ordered inverted index. During retrieval, thelive items are retrieved from the second inverted index and returnedfirst to insure that the live items are returned before other items fromthe ordered inverted index.

According to one aspect of the invention, further efficiency inretrieval of items from an inverted index may be achieved by storing theindexed items' file grouping parameters separately, outside the invertedindex. During later retrieval, an item's file grouping parameters can beread very quickly, and only query results for selected groups need beevaluated further.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a block diagram overview of an architecture for processing aninverted index according to one exemplary embodiment of the invention.

FIG. 2 is a block diagram illustrating one aspect of processing anordered index according to one exemplary embodiment of the invention.

FIG. 3 is a block diagram illustrating another aspect of processing anordered index according to one exemplary embodiment of the invention.

FIGS. 4-5 are flow diagrams illustrating certain aspects of performing amethod of processing an ordered index according to one exemplaryembodiment of the invention.

FIG. 6 is a block diagram overview of an exemplary embodiment of a dataprocessing system, which may be a general purpose computer system andwhich may operate in any of the various methods described herein.

DETAILED DESCRIPTION

The embodiments of the present invention will be described withreference to numerous details set forth below, and the accompanyingdrawings will illustrate the described embodiments. As such, thefollowing description and drawings are illustrative of embodiments ofthe present invention and are not to be construed as limiting theinvention. Numerous specific details are described to provide a thoroughunderstanding of the present invention. However, in certain instances,well known or conventional details are not described in order to notunnecessarily obscure the present invention in detail.

The present description includes material protected by copyrights, suchas illustrations of graphical user interface images. The owners of thecopyrights, including the assignee of the present invention, herebyreserve their rights, including copyright, in these materials. Thecopyright owner has no objection to the facsimile reproduction by anyoneof the patent document or the patent disclosure, as it appears in thePatent and Trademark Office file or records, but otherwise reserves allcopyrights whatsoever. Copyright Apple Computer, Inc. 2007.

Various different software architectures may be used to implement thefunctions and operations described herein, such as to perform themethods shown in FIGS. 4-5. The following discussion provides oneexample of such an architecture, but it will be understood thatalternative architectures may also be employed to achieve the same orsimilar results. The software architecture 100 shown in FIG. 1 is anexample which is based upon the Macintosh operating system. Thearchitecture 100 includes indexing software 102 and an operating system(OS) kernel 124 which is operatively coupled to the indexing software102, as well as other software programs, such as find by contentsoftware 106 and find by metadata software 110 (which may be the Finderprogram referenced earlier), and other applications not shown.

In one exemplary embodiment, the find by content software 106 and/or thefind by metadata software 110 is used to find a term present in the filedata 104 or meta data 108. For example, the software 106/110 may be usedto find text and other information from word processing or textprocessing files created by word processing programs such as MicrosoftWord, etc.

The find by content software 106 and find by metadata software 110 areoperatively coupled to databases which include one or more indexes 122.The indexes 122 represent at least a subset of the data files in astorage device, including file data 104 and meta data 108, and mayinclude all of the data files in a particular storage device (or severalstorage devices), such as the main hard drive of a computer system. Theone or more indexes 122 comprise an indexed representation of thecontent and/or metadata of each item stored on the data files 104/108,such as a text document, music, video, or other type of file. The findby content software 106 searches for a term in that content by searchingthrough the one or more index files 122 to see if the particular term,e.g., a particular word, is present in items stored on data files 104which have been indexed. The find by content software functionality isavailable through find by metadata software 110 which provides theadvantage to the user that the user can search the indexes 122 for thecontent 104 within an item stored on the data files 104 as well as anymetadata 108 that may have been generated for the item.

In one embodiment of the present invention, indexing software 102 isused to create and maintain the one or more indexes 122 that areoperatively coupled to the find by content and metadata softwareapplications 106/110. Among other functions, the indexing software 102receives information obtained by scanning the file data 104 and metadata 108, and uses that information to generate a postings list 112 thatidentifies an item containing a particular term, or having metadatacontaining a particular term. As such, the postings list 112 is a typeof inverted index that maps a term, such as a search term, to the itemsidentified in the list. In a typical embodiment, the informationobtained during the scan includes a unique identifier that uniquelyidentifies the item containing the particular term, or having metadatacontaining the term. For example, items such as a word processing ortext processing file have unique identifiers, referred to as ITEMIDs.The ITEMIDs are used when generating the postings list 112 to identifythose items that contain a particular term, such as the word “Apple.”ITEMIDs identifying other types of files, such as image files or musicfiles, may also be posted to the postings list 112, in which case theITEMID typically identifies items having metadata containing aparticular term.

In one embodiment, the indexing software 102 accumulates postings lists112 for one or more terms into one or more update sets 120 and, fromtime to time, flushes the updates sets 120 into one or more index files122. The postings lists 112 for one or more items may also be stored ina postings file 118. The indexing software 102 may employ one or moreindexing tables 114 that comprise one or more term tables, including atwo-level table that separates the more frequently occurring terms fromthe less frequently occurring terms. The tables 114 may also include apostings table that comprises one or more postings lists for the termsthat are being indexed. In one embodiment, the indexing software maymaintain a live index 116 to contain the most current index. In somecases, updates to an index may be generated in a delta postings list 126that is a specially marked postings list that may be dynamically appliedto an index 122, postings files 118, updates sets 120, or other forms ofan index in order to insure that the most current information isreturned whenever those indexes are accessed.

FIG. 2 is a block diagram illustrating one aspect of processing anordered index according to one exemplary embodiment of the invention. Asshown, a postings list 112 is ordered into a desired retrieval order 202resulting in a sorted postings list 112A. Typically, the desiredretrieval order 202 is the order in which the items contained in thelist were last accessed, modified, or received, i.e., a reverse timeorder, with more recently accessed items first and less recentlyaccessed items last. However, other retrieval orders may be specified,such as the alphabetical order of the name of the item.

In a typical embodiment, indexing software 102 is operatively coupled tothe sorted postings list 112, which serves as an input to the indexingsoftware 102 during the creating or updating of an ordered index file204. The ordered index file 204 is an inverted index that is createdfrom items that have been posted or sorted into a desired retrievalorder. Accordingly, the indexing software 102 inserts the items from thesorted postings list 112A into the ordered index file 204 in the desiredretrieval order. The indexing software 102 also maintains a live index116 that contains the most recent updates to the ordered index 204.

In one embodiment, the indexing software 102 stores each indexed item'sfile grouping parameters in a separate store 206. An item's filegrouping parameters have values that indicate certain characteristics orother aspects of an item, such as a category into which the item falls,e.g., text, movie, music, etc. and/or the name of the item, etc. One ofthe parameter values may also indicate a threshold limit value, referredto as a TOP-N value, to be employed when retrieving items in aparticular category. For example, the threshold limit value may be 5items, meaning that, during retrieval from the ordered index 204, only 5items should returned in any one category, such as only 5 movies, beforereaching the TOP-5 limit. Other types of parameters may also be storedin the file grouping parameters store 206.

FIG. 3 is a block diagram illustrating another aspect of processing anordered index according to one exemplary embodiment of the invention. Asshown, the ordered index file 204 is operatively coupled to retrievalsoftware 302 and an indexed item's file grouping parameters 206. Theretrieval software operates to retrieve items from the ordered indexfile 204 in the desired retrieval order 202 and to place the retrieveditems into a query result 304. In one embodiment, the retrieval software302 further operates to interrogate the indexed item's file groupingparameters 206 and to filter the query result 304 in accordance with theindexed item's file grouping parameters 206, including limiting theretrieval to the top-N search results in a particular category, such asa movie, where the “N” value and the “movie” category value are bothexamples of file grouping parameters 206. Thus, a file groupingparameter may indicate a category of the item being retrieved, e.g., adocument, a movie, a presentation, etc., as well as an upper limit orthreshold, referred to as a top-N limit. By setting a top-N limit foreach category, retrieval of items in one category does not overwhelm theother categories. This results in better performance during retrievaland is more user-friendly.

In one embodiment, the retrieval software 302 is also operativelycoupled to a live index 116, in which the most recent updates to anindex are maintained. During retrieval, the retrieval software mayoptionally retrieve items from the live index 116 prior to attemptingretrieval of items from the ordered index file to insure that only themost recent results are returned in the query result 304. In oneembodiment, the retrieval from the live index 116 and ordered index 204may be merged together to form the query result 304 that is returned tothe user.

FIG. 4-5 are flow diagrams illustrating certain aspects of performing amethod of processing an ordered index according to one exemplaryembodiment of the invention. In FIG. 4, the method to be performedbegins at block 402, in which items in a postings list are ordered intoa desired retrieval order. The items are typically ordered as they areposted into the postings list, but may also be ordered after posting tothe postings list. At processing block 404, the method continues withinserting items into the ordered index in the desired retrieval order.At block 406, the method continues by optionally generating a separate“live” index for the most recently updated items, referred to as “live”items. Lastly, at block 408, the method stores the item's file groupingparameter(s) for each item that is updated to the ordered index.

In FIG. 5, the method begins at block 502, in which retrieval softwarethat is operatively coupled to an ordered index receives a query from arequestor. At block 504, the retrieval software optionally firstretrieves items matching the query from a live index. At block 506, theretrieval software retrieves items matching the query from the orderedindex. The results from blocks 504 and 506 may be merged as needed,after which, at block 508 the retrieval software commences filtering theitems in the query results based on the retrieved items' file groupingparameters. For example, the filter may allow only the first 5 movies tobe returned in a query result set having more than 5 movie matches.Finally, at block 510, the retrieval software relays the filteredresults back to the requestor.

FIG. 6 illustrates an example of a typical computer system which may beused with the present invention. Note that while FIG. 6 illustratesvarious components of a computer system, it is not intended to representany particular architecture or manner of interconnecting the componentsas such details are not germane to the present invention. It will alsobe appreciated that network computers and other data processing systemswhich have fewer components or perhaps more components may also be usedwith the present invention. The computer system of FIG. 6 may, forexample, be a Macintosh computer from Apple Computer, Inc.

As shown in FIG. 6, the computer system 600, which is a form of a dataprocessing system, includes a bus 602 which is coupled to amicroprocessor(s) 603 and a ROM (Read Only Memory) 607 and volatile RAM605 and a non-volatile memory 606. The microprocessor 603 may be a G3 orG4 microprocessor from Motorola, Inc. or one or more G5 microprocessorsfrom IBM. The bus 602 interconnects these various components togetherand also interconnects these components 603, 607, 605, and 606 to adisplay controller and display device 604 and to peripheral devices suchas input/output (I/O) devices which may be mice, keyboards, modems,network interfaces, printers and other devices which are well known inthe art. Typically, the input/output devices 609 are coupled to thesystem through input/output controllers 608. The volatile RAM (RandomAccess Memory) 605 is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. The mass storage 606 is typically a magnetic hard drive or amagnetic optical drive or an optical drive or a DVD RAM or other typesof memory systems which maintain data (e.g. large amounts of data) evenafter power is removed from the system. Typically, the mass storage 606will also be a random access memory although this is not required. WhileFIG. 6 shows that the mass storage 606 is a local device coupleddirectly to the rest of the components in the data processing system, itwill be appreciated that the present invention may utilize anon-volatile memory which is remote from the system, such as a networkstorage device which is coupled to the data processing system through anetwork interface such as a modem or Ethernet interface. The bus 602 mayinclude one or more buses connected to each other through variousbridges, controllers and/or adapters as is well known in the art. In oneembodiment the I/O controller 608 includes a USB (Universal Serial Bus)adapter for controlling USB peripherals and an IEEE 1394 controller forIEEE 1394 compliant peripherals.

It will be apparent from this description that aspects of the presentinvention may be embodied, at least in part, in software. That is, thetechniques may be carried out in a computer system or other dataprocessing system in response to its processor, such as amicroprocessor, executing sequences of instructions contained in amemory, such as ROM 607, RAM 605, mass storage 606 or a remote storagedevice. In various embodiments, hardwired circuitry may be used incombination with software instructions to implement the presentinvention. Thus, the techniques are not limited to any specificcombination of hardware circuitry and software nor to any particularsource for the instructions executed by the data processing system. Inaddition, throughout this description, various functions and operationsare described as being performed by or caused by software code tosimplify description. However, those skilled in the art will recognizewhat is meant by such expressions is that the functions result fromexecution of the code by a processor, such as the microprocessor 603.

1. A machine-implemented method for preparing an inverted index forstorage on a storage medium, the method comprising: scanning itemscontaining terms from which an ordered inverted index is to be prepared;generating a postings list of scanned items containing a particularterm; ordering the scanned items in the postings list in a retrievalorder; inserting the scanned items from the postings list into theordered inverted index in the retrieval order; retrieving the scanneditems from the ordered inverted index in the retrieval order; preparinga live inverted index of live items not previously scanned, wherein thelive inverted index is separate from the ordered inverted index ofscanned items; and retrieving the live items from the live invertedindex before retrieving scanned items from the ordered inverted indexfor a reguestor via an output device.
 2. A machine-readable storagemedium having instructions, when executed by a machine, cause themachine to perform a method for method for preparing an inverted indexfor storage on a storage medium, the method comprising: scanning itemscontaining terms from which an ordered inverted index is to be prepared;generating a postings list of scanned items containing a particularterm; ordering the scanned items in the postings list in a retrievalorder; inserting the scanned items from the postings list into theordered inverted index in the retrieval order; retrieving the scanneditems from the ordered inverted index in the retrieval order; preparinga live inverted index of live items not previously scanned, wherein thelive inverted index is separate from the ordered inverted index ofscanned items; and retrieving the live items from the live invertedindex before retrieving the scanned items from the ordered invertedindex to a requestor via an output device.